Skip to content

perf(explore): rank full-text matches before reading their rows - #259

Merged
bompus merged 1 commit into
fork/consolidatedfrom
perf/explore-fts
Sep 28, 2026
Merged

bompus merged 1 commit into
fork/consolidatedfrom
perf/explore-fts

Conversation

@bompus

@bompus bompus commented Sep 28, 2026

Copy link
Copy Markdown
Owner

Follow-up to #258. After it, the full-text search was the largest remaining explore statement.

searchNodesFTS joined nodes on nodes_fts.id = nodes.id and sorted every match, with all its node columns, by bm25. In an external-content FTS table id is read back from nodes, and the join then looked it up again through the id index. The query now ranks (rowid, score) in a subquery (still joined to nodes by rowid for the kind and language filters) and reads full rows only for the page it returns. Ties order by rowid, as the old form did implicitly.

Measured on trezor-suite (151k nodes), recorded as ledger §5.100:

Before After
8 FTS queries, explore's 21-kind filter 84.3 ms 36.7 ms
Same, 8 definition kinds 69.9 ms 24.3 ms
10-question explore round median, paired runs 4,776 / 4,797 ms 4,469 / 4,558 ms
Explore output byte-identical on all 10 questions

Checks: full suite 6,125 passed, 32 skipped; readme-sync.

README rows checked: the "About this fork" table and measured sections. No row quotes single-call explore latency, so none changed. The Bun concurrent-explore line was already marked for re-measurement in #258.

searchNodesFTS sorted every match with all its node columns and joined
through the id column of the external-content FTS table. It now ranks
(rowid, score) in a subquery joined by rowid and reads full rows only for
the page it returns. Ties still order by rowid.

trezor-suite: FTS step 2.3-2.9x faster; explore round about 6% faster;
output byte-identical.
@bompus
bompus merged commit eb2a24f into fork/consolidated Sep 28, 2026
@bompus
bompus deleted the perf/explore-fts branch September 28, 2026 08:06
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant